DICE API Documentation/com.embabel.dice.incremental

Package-level declarations

Types

abstract class AbstractIncrementalAnalyzer<T, R>(historyStore: ChunkHistoryStore, formatter: IncrementalSourceFormatter<T>, config: WindowConfig = WindowConfig(), contentHasher: ContentHasher = Sha256ContentHasher) : IncrementalAnalyzer<T, R>

Handy base implementation of IncrementalAnalyzer providing:

AnalysisBookmark

data class AnalysisBookmark(val sourceId: String, val endIndex: Int, val processedAt: Instant)

Resume marker for incremental source analysis within a ContextId.

BookmarkKey

data class BookmarkKey(val contextId: <Error class: unknown class>, val sourceId: String)

Composite key for per-source analysis bookmarks within a ContextId.

ChunkHistoryStore

interface ChunkHistoryStore

Tracks the history of processed chunks for incremental analysis. Enables deduplication and resumption of processing.

ConversationSegmenter

class ConversationSegmenter(maxSize: Int = DEFAULT_MAX_SIZE, settle: Duration = DEFAULT_SETTLE, minItems: Int = 1)

Splits a timestamped item stream (e.g. a chat channel's messages) into conversation segments, so each segment can be handed to IncrementalSource-based extraction as one conversation.

ConversationSource

class ConversationSource(conversation: <Error class: unknown class>) : IncrementalSource<<Error class: unknown class>>

Adapts a Conversation to the IncrementalSource interface for incremental analysis.

HashKey

data class HashKey(val contextId: <Error class: unknown class>, val contentHash: String)

Composite key for content-hash deduplication within a ContextId.

IncrementalAnalyzer

interface IncrementalAnalyzer<T, R>

Analyzes incremental sources (conversations, listening history, etc.) and produces results of type R.

IncrementalSource

interface IncrementalSource<T>

Represents a source of items that grows over time. Used for incremental processing of conversations, listening history, etc.

IncrementalSourceFormatter

interface IncrementalSourceFormatter<T>

Formats items from an IncrementalSource into text for processing.

InMemoryChunkHistoryStore

class InMemoryChunkHistoryStore : ChunkHistoryStore

Thread-safe in-memory implementation for testing. Not intended for production use.

MessageFormatter

class MessageFormatter : IncrementalSourceFormatter<<Error class: unknown class>>

Formats conversation messages for proposition extraction.

ProcessedChunkRecord

data class ProcessedChunkRecord(val bookmarkKey: BookmarkKey, val hashKey: HashKey, val startIndex: Int, val endIndex: Int, val processedAt: Instant)

Record of a processed chunk, keyed by BookmarkKey and HashKey.

WindowConfig

data class WindowConfig(val windowSize: Int = 20, val overlapSize: Int = 2, val triggerInterval: Int = 4)

Configuration for windowed processing.